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Abstract 

Sixty years ago, S. B. Myers and N. E. Steenrod (Ann. of Math. 40 (1939), 400-416) 
showed that the isometry group of a Riemannian manifold without boundary has a 
structure of Lie group. Recently A. V. Bagaev and N. I. Zhukova (Siberian Math. J. 
48 (2007), 579-592) proved the same result for a Riemannian orbifold. In this paper, we 
firstly show that the isometry group of a Riemannian manifold M with boundary has 
dimension at most \ dim M (dim M. — 1). Then we completely classify such Rieman- 
nian manifolds with boundary that their isometry groups attain the preceding maximal 
dimension. 
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1 Introduction 

Let M be a connected smooth Riemannian manifold with or without boundary. A priori, 
there exists two definitions of isometry on M. The first one is given above as we think M 
of as a metric space. The second is denned to be a diffeomorphism of M onto itself which 
preserves the metric tensor. In case of Riemannian manifolds without boundary, these two 
definitions are equivalent according to Myers and Steenrod [8 J in 1939 (see also pp. 169-172 
in Kobayashi-Nomizu [6] for a proof). Moreover, Myers and Steenrod [8 J proved the follow- 
ing result on the isometry group of a Riemannian manifold without boundary. 

Fact 1.1. Let M be a connected smooth Riemannian manifold without boundary. Then 
the isometry group I(M) is a Lie transformation group with respect to the compact- 
open topology. For each x € M, the isotropy subgroup I X (M) is compact. If M is 
compact, I(M) is also compact. 

S. Kobayashi (see p. 15 and p. 41 in |5J) gave a different proof to Fact 1.1 by the 
concept of G-structure from the original one by Myers-Steenrod. Furthermore Kobayashi 
proved that there exists a natural embedding of the isometry group I(M) into the orthonor- 
mal frame bundle O(M) of M such that I(M) becomes a closed submanifold of O(M). It 
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is this submanifold structure that makes I(M) into a Lie transformation group. Following 
this idea, he further proved 

Fact 1.2. (see pp. 46-47 in j5]) Let M be an n- dimensional connected Riemannian 
manifold without boundary. Then the isometry group I(M) has dimension at most 
(n + 1)n/2. // dim I(M) = n(n + 1)/2, then M is isometric to one of the following 
spaces of constant sectional curvature: 

(a) An n- dimensional Euclidean space R n . 

(b) An n- dimensional unit sphere S n = {x e R n+1 : |x| = 1} in R n+ . 

(c) An n- dimensional projective space RP n = S n /{±1}- 

(d) Tin u- dimensional, simply connected hyperbolic space H n of constant sectional cur- 
vature —1 . 

As long as the isometry group of a Riemannian orbifold is concerned, quite recently, A. 
V. Bagaev and N. I. Zhukova pQ showed the same result as Facts 1.1-2. They generalized the 
idea of Kobayashi to their setting by using the orthonormal frame bundle of a Riemannian 
orbifold. In this paper we consider a special class of orbifolds — manifolds with boundary. 
We firstly observe that the dimension of the isometry group I(M) of a Riemannian manifold 
M with boundary does not exceed \ dim M (dim M— 1 ). Then we classify such Riemannian 
manifolds M with boundary that the isometry groups I(M) attain the preceding maximal 
dimension. We divide the lengthy classification list into three parts: Theorems 1.1-3, due 
to that their proofs will use different ideas. The notations in Fact 1.2 will be used in the 
following theorems. 

Theorem 1.1 Let M be an n- dimensional compact, connected smooth Riemannian 
manifold with boundary and u > 2. Suppose that the isometry group I(M) is of di- 
mension n(n— 1 )/2. Then M is diffeomorphic to either of the following four manifolds : 
the closed n-dimensional unit ball D n = {x e R n : |x| < 1} in R n , the two cylinder-like 
manifolds S n_1 x [0, 1] and RP n_1 x [0, 1], and the manifold RP n \U constructed from 
RP n with an n-dimensional open disk U c RP n removed, where the closure U in RP n 
is diffeomorphic to D n . Furthermore, we can characterize the metric tensor gjvi of M 
as follows : 

(1) IfM is diffeomorphic to B n , then the metric qm. of M is rotionally symmetric with 
respect to a unique interior point O of M. That is, qm. can be expressed by qm = 
dt 2 + cp 2 (t) g cn- 1 . where Qs^-t is the standard metric on the unit sphere S n 1 C R n 
and the function cp : (0, R] — > (0, oo) is smooth, cp(0) = 0, and 

<p(«wO(0) =0, cp(0) = 1. 

(2) // M is diffeomorphic to S n_1 x [0, 1], then the metric 9m can be expressed by 
dt 2 + f 2 (t) g S n-i , where T is a positive number and f is a positive smooth function on 
[— T/2, T/2]. The similar statement holds for M diffeomorphic to RP n_1 x [0, 1]. 

(3) Suppose that M is diffeomorphic to RP n \U. Then we can find a Riemannian 
manifold M' = S n x [—T/2, T/2] endowed with the metric dt 2 + f 2 (t) g S n-i , where 
f : [—T/2, T/2] — ) (0, oo) is an even smooth function, and an involutive isometry (3 
of M' defined by (3(x, t) = (— x, — t) such that M is the quotient space of M' by the 



group {1 , |3}. 

Theorem 1.2 Let M be a noncompact connected Riemannian manifold with boundary 
3M and of dimension n > 2 such that dim I(M) = n(n — 1)/2 and 3M has at least one 
compact component. Then M is diffeomorphic to either S n_1 x [0, 1) or RP n_1 x [0, 1). 
In the former case, the metric of M can be expressed by dt 2 + f 2 (t) g S n-i , where 
f : [0, T) — > (0, oo) is a smooth function and J is a positive number or oo. Moreover 
M 45 complete if and only if T = oo. T/ie similar statement holds for the latter case. 

Theorem 1.3 Let M be a connected Riemannian manifold with noncompact boundary 
3M and of dimension n > 2 5uc/i that dim I(M) = n(n — 1)/2. Denote &y H k , k > 2, 
i/ie k- dimensional complete simple connected Riemannian space of constant sectional 
curvature — 1. Then M is diffeomorphic to either R n 1 x [0, 1] or R n_1 x [0, 1). Fur- 
thermore, we can characterize the metric tensor qm of M as follows: 

(1) // M is diffeomorphic to R n_1 x [0, 1], then there exists a positive number T and 
a smooth function f : [0, T] — > (0, oo) such that the metric tensor qm on M can be 
expressed by 9m = dt 2 + f 2 (t)g Rn -i or gjvi = dt 2 + f 2 (t)g H n-i with t € [0, T]. Of course, 
we identify H 1 with R 1 . The metric qm in this case is always complete. 

(2) // M is diffeomorphic to R n 1 x [0, 1), then there exists a number T € (0, oo] and 
a smooth function f : [0, T) — > (0, oo) such that the metric tensor qm on M can be 
expressed by gjvi = dt 2 + f 2 (t)g Rn -i or gjvi = dt 2 + f 2 (t)g H n-i with t € [0, T). Moreover, 
the metric 9m is complete if and only if T = oo. 

This paper is organized as follows. In section 2, we prove the fact that the two definitions 
of isometry coincide on Riemannian manifolds with boundary (see Proposition 2.1). It seems 
that Bagaev-Zhukova pQ did not mention this fact in their setting of Riemannian orbifolds. 
The idea of making reduction to the boundary in the proof of Proposition 2.1 will be used 
many times afterwards. In this section we also show the above mentioned observation that 
the isometry group I(M) of a Riemannian manifold M with boundary is a Lie transformation 
group of dimension at most \ dim M (dim M — 1 ) (see Theorem 2.1). Although our proof of 
the observation is based on the idea of Proposition 2.1, to avoid the troublesome argument 
of point set topology, we also use the result in Bagaev-Zhukova [lj that I(M) has a Lie 
group structure. In sections 3 through 5, we use the metric geometry and the theory of 
transformation group to prove Theorems 1.1 through 1.3. 

2 Some properties of isometry group 

In the following sections we always let M be an n-dimensional connected, smooth Rieman- 
nian manifold with boundary and n > 2. With the induced metric from M, the boundary 
3M of M is an (n— 1 )-dimensional Riemannian manifold without boundary. Note that 3M 
has at most countable connected components. Consider a diffeomorphism cj) of M onto itself 
which preserves the metric tensor. If p is an interior point of M, then <$> maps p to another 
interior point, say q, and the differential map Dcf) at p induces an orthogonal transform 
from the tangent space at p to the one at q. If u is a point on the boundary 3M, then the 



tangent space T U M. at u should be thought of as the upper half space 

{x = (xt , x 2 , • • • , x n ) € R n : x n > 0} 
of the Euclidean space R n . That is, 

T-uM = TJ3M) + {An u : A > 0}, 

where n u is the inner unit normal vector at u. Since cf) maps u to another point v on 3M, 
the differential map Dc|) at u maps n u to n v , and maps T u (dM) orthogonally onto T V (3M). 
Hence, in this sense, we may also call that the differential map D<{) at u is an orthogonal 
transform from T U M onto T V M. Hence (J) leaves the boundary 3M invariant and induces an 
isometry of 3M. Remember that by Myers-Steenrod [8] the two definitions of isometry on 
3M are equivalent. 

Let d(-, •) be the distance function on M induced by the metric tensor of M and i|) a 
bijection on M which preserves d(-, •)• Since 4> is a homeomorphism of M onto itself, its 
restrictions to the boundary 3M is a homeomorhism onto itself, so is the restriction to the 
interior of M. In fact, we have a stronger property about i[> in the following 

Proposition 2.1. A distance-preserving bijection i[> of M is a diffeomorphism which 
preserves the metric tensor of M. That is, the two definitions of isometry of M are 
equivalent. 

Proof. We firstly consider the property of tJj near a point p of in the interior Int(M) of 
M. There exists an open neighborhood U c Int(M) of p such that the restriction 4>|u of 
4> to U is a distance-preserving map onto the open neighborhood V = ih(U) C Int(M) of 
q = ip(p). Since the two definitions of isometry for Riemannian manifolds without boundary 
are equivalent (see pp. 169-172 in Kobayashi-Nomizu [6] for a proof), ih|u : U — > V is a 
diffeomorphism preserving the metric tensor. Hence, 4>lint{M) : Int(M) — » Int(M) is also a 
diffeomorphism preserving the metric tensor. It suffices to prove that for each point p € 3M, 
4> is smooth near p and D4> at p is an orthogonal transform from T P M onto T q M, where 
q = iMp) 6 3M. We divide the proof into two steps. 

Step 1 Remember that n p denotes the inner unit normal vector at p. Choose 6 > so 
small that the geodesic y(p, t) := exp p (tn p ), t e [0, 6], satisfies 

d(y(p, t), 3M) = d(y(p, t), y(p, 0)) = t. (1) 

Since the geodesic emanating from each point with this property is unique, the image i|)oy 
of y under the isometry i|> is also a geodesic perpendicular to the boundary 3M at the 
initial point q. Actually, we will see later that the differential D\|> at p maps n p to n q . In 
3M, we choose a small open neighborhood V C 3M of p such that for each point p'eV 
the geodesic y(p', t), t € [0, 6/2], satisfies ([T]). Then the map y(-, t) : p' i— > y(p', t) gives a 
diffeomorphism of V onto a hypersurface V t in Int(M) for each t e (0, 6/2]. We can define 
the similar map from the neighborhood 4>(V) c 3M of q and denote the map also by y(-, t). 
We observe that 

^ °y{; t) =y{; t) oi|) 

holds for each p € V and each t G [0, 6/2] so that y(tKV), t) = xp(V t ). Hence the map tp|v 
can be thought of as the composition of three diffeomorphisms, 

^M-) = (y(-,t))^oxl;oy(., t). 



So ib|y is a diffeomorphism of V onto iMV). Since tp|v t is an isometry of V t onto 4>(V t ) for 
each < t < 5/2, letting t — > 0, we can see that 4>|v is a one-to-one distance-preserving map 
of V onto iMV). Since the two definitions of isometry for Riemannian manifolds without 
boundary are equivalent, \|)|y is a diffeomorphism onto ij>(V) preserving the metric tensor. 

Step 2 By Step 1 and its preceding argument, we can take a small open neighborhood 
U c M of p such that the exponential map exp p at p is a diffeomorphism from some 
neighborhood U of in T P M onto U. Remember that the partial derivative of ij; exists 
at the direction of the inner unit normal n p , and equals n q . That is, the differential 
map D4> at each p in U n 3M is an orthogonal transform from T p M to T^( p )M. Since 
4> is a homeomorphism, we may assume that iMU) is contained in a normal coordinate 
neighborhood of q. Since both exp p and exp q are local diffeomorphisms, the equality 

4> o exp p = exp q o(Dt|;(p)) 

in U, which gives that 4>|u is diffeomorphism onto i|)(U). □ 

The above proof essentially follows the idea in pp. 169-172 of Kobayashi-Nomizu [6]. We 
repeat it here because its idea and notations will be used many times later. The following 
lemma is elementary and useful, but its proof is omitted. 

Lemma 2.1. Let <t be an isometry of M. // <b has a fixed point p € M and the 
differential Dcj) at p is the identity map of T P M, then <b is the identity map of M. 

Proposition 2.2. (1) Let $ be an isometry of M. Then the restriction 4>|aM to the 
boundary 3M is an isometry o/3M. Moreover, if $ leaves each point of a component 
B o/3M fixed, then <t is the identity map of M. 

(2) Let be an element in the identity component I (M) of the isometry group of M. 
The restriction 4>|b to a connected component B of 3M is an element in the identity 
component I°(B) of the isometry group o/B. This map 

i:I°(M)->I°(B), cp ^ cp| B 

gives a continuous monomorphism with the image closed in I°(B). That is, 1 is a 
regular embedding of the Lie group I (M) into the Lie group I°(B). 

Proof. 

(1) The first statement have been shown in the proof of Proposition 12.11 If each point p of 
a component B of 3M is fixed by <b, then the differential D<b at p is the identity map of 
T P M. By Lemma I2TT1 <t> is the identity map of M. 

(2) Given a point p in a connected component B of 3M, we claim that each element cj) in 
I°(M) maps p to a point in B. Otherwise, we assume that cb(p) lies in another component 
B' distinct from B. Choosing a path {cb t } in I°(M) with <bo = idjw and cj? i = 4>, we obtain a 
path {4>t(p)} connecting p and cb(p). Since each diffeomorphism of M maps 3M onto 3M, 
we find that the path {cj>t} lies on 3M and reach a contradiction. By the proof of Proposition 
I2.1[ we know that 4>|b is an isometry of B, which is actually an element of I°(B). Moreover, 
if c()|b is the identity map of B, then by (1) cf) is the identity map of M. Thus, the map 
1 : I°(M] — ) I°(B), c}) 1-4 4>|b gives a continuous monomorphism of I°(M) into I°(B). 



Finally we need to show that the image of I°(M) under i is closed in I (B) with respect 
to the compact-open topology. We divide the proof into two steps. 

Step 1 We show that for a sequence {<$> n } of isometries in I (M.) such that 

cb n | B ^id B in I°(B), 

there holds 4> n — > idjvi in I°(M). By Fact 1.1, I°(B) has the structure of a Lie group, whose 
topology from its smooth structure coincides with the compact-open topology. We may 
assume that 4> n |B — > ids in the C 1 topology of I°(B). That is, cb n converges to the identity 
map idK in the sense of the C 1 norm in each compact neighborhood K in the topological 
space B. Then, by using the normal coordinate charts with respect to each point in K (see 
the last three lines of the proof of Proposition 2.1), we find that there exists a compact 
neighborhood K of M. such that Kc/C and cb n (p) converges uniformly to p for each point 
p € K. 

Step 2 We show that if a sequence {<t> n } of isometries in I°(M) satisfies 4> n |B — > <b 
in I°(B), then there exists xp £ I°(M) such that 4>|b = 4>- Since, by Myers-Steenrod [8] 
and Bagaev-Zhukova [lj, both I°(B) and I°(B) have structures of Lie groups, they can be 
endowed with a Riemannian metric. By the Cauchy criterion, 4> n |B converges in I°(B) 
4) n | B (c() m | B ) _1 converges to id B as m, n — > oo. Step 1 tells us that cb n (4> m ) _1 converges to 
idjvi as m, n — > oo. That is, cb n converges to some 4> € I°(M) such that 4>| B = 4>- D 

As an immediate corollary of Proposition 2.2 and Fact 1.3, we obtain 

Theorem 2.1. The isometry group I(M n ) has a structure of Lie group of dimension 
at most n(u — 1 )/2. 



3 Proof of Theorem 1.1 

Let M be a Riemannian manifold satisfying the condition of Theorem 1.1 in this section. By 
Proposition 2.2 and Fact 1.2, the isometry group 1(B) of each component B of 3M attains 
the maximal dimension n(n— 1 )/2, so B is isometric to either S n_1 or RP n_1 with constant 
sectional curvature 1. If n = 2, the 3M consists of circles. But our argument later also goes 
through in this case. 

Suppose that there exists a component B isometric to the sphere S n_l . By the proof 
(see pp. 46-47 in [5]) of Fact 1.2, G := I°(M) is isomorphic to SO(n) and its action on B is 
just the linear action of SO(n) on S n_1 . We may identify G with SO(n), B with S n_1 up 
to a scaling of metric. Remember that G acts transitively on S n ~ 1 and the isotropy group 
G x at each point x of S n_1 is isomorphic to H := SO(n — 1). Here we use the notation in 
the proof of Proposition 2.1. Choose a positive number 6 > such that the map y{-, t) 
is a diffeomorphism of B onto a hypersurface B t in Int(M) for each t € (0, 5]. Since the 
G-action interchanges with y(-, t), G leaves each B t invariant. We claim that the G-action 
on M has the principal orbit whose type is SO(n)/SO(n — 1 ). Remember that the union of 
the principal orbits of type different from G/H forms an open and dense subset of M. For 
the detail of this, see Theorem 4.27 in pp. 216-220 of Kawakubo [4J, where only manifolds 



without boundary are considered. But Theorem 4.27 in Kawakubo [4j also holds for our 
case by virtue of the map y(-, t). The claim follows from that the union M(H) of orbits 
with type G/H contains the open subset U tg [ ,5]Bt in M. Therefore, every component of 
3M is isometric to S 11-1 . By Theorem 4.19 (pp. 202-203) and Theorem 4.27 in Kawakubo 
[I], the orbit space M(H)/G is a connected smooth 1-manifold, whose boundary coincides 
with the orbit space 3M/G. Hence, 9M has at most two components. We also have the sim- 
ilar argument when 3M has a component isometric to RP n_1 . Summing up, we have proved 

Lemma 3.1 3M has at most two components, each of which is isometric to either 
S n_1 or RP n_1 . Moreover, if 3M has two components, then the two components are 
isometric up to a scaling of metric. 

Lemma 3.2 Let 3M have two components. Then M is diffeomorphic to either S n_1 x 
[0, 1] or RP n_1 x [0, 1]. IfM is diffeomorhic to S n_1 x [0, 1], then there exists a positive 
number T and a positive smooth function f (t) on [0, T] such that the Riemannian metric 
qm on M can be expressed by 

g M = dt 2 + f 2 (t)g S n-,. 
The similar statement holds for M diffeomorphic to RP n_1 x [0, 1]. 

Proof. We only prove the case that both the two component of 3M are isometric to S n_1 . 
By the proof of Lemma 3.1, the orbit space M(H)/G is a closed interval. On the other 
hand, the orbit space M(H)/G is a dense subset in the total orbit space M/G. Therefore, 
M = M(H), i.e. all the orbits of G-action on M is of principal type. So M is a smooth 
fiber bundle with fiber S n_1 over a compact interval. Actually M is diffeomorphic to the 
product of S n_1 and a compact interval. 

We use the notation in the proof of Proposition 12.11 in what follows. Let B and B' be 
the components of 3M. The two spheres B and B' may have different size. Choose a point 
p G B. Then we claim that the geodesic y(p, t) with initial velocity n p terminates at some 
point p'GB' and with the ending verlocity n p / at a positive time determined later. Indeed, 
choosing p' G 3M such that d(p, B') = d(p, p') =: T(p), we can find a geodesic between p 
and p' whose length is T(p). It is clear that this geodesic is perpendicular to the boundary 
3M at p'. It also is perpendicular to 3M at p. Otherwise, we can find another point q on 
B and a path I connecting q and p' of length less than T(p). Choosing an element a € G 
mapping q to p, we obtain a path a(£) connecting p and a(p') € B' of length less than 
T(p). Contradiction! Therefore, this geodesic is exactly the one {y(p, t) : t e [0, T(p)]} in 
the claim. We claim again that T(p) equals the distance T bewteen B and B'. Actually, 
there exists po such that the geodesic y(po, [0, T(po)]) satisfies 

T(po) = d(p , y(po, T( Po )) =d(B, B'). 
But G acts transitively on the set of geodesies 

{y(p,[0, T( P )]):peB} 

of geodesies connecting B and B', which implies that these geodesies have the same length 
T. So each point p in B has the same distance T to B'. 



We claim that the map y(-, t) : B — > M is a diffeomorphism of B onto the hypersurface 
B t for each t € [0, T]. Similarly as the preceding paragraph, this map is sujective. It is also 
injective. Otherwise, there exists < T' < T and two distinct points pi and P2 in B such 
that y(pi, T') =y(p2> T') =: q. Then we get a curve y(pi, [0, T'])Uy(p2, \J', T]) irregular 
at point q with length T connecting B and B'. Contradiction. Since this map is equivariant 
with respect to the G actions on B and B t , every point of B t is a regular point of the map by 
the Sard theorem. Combining above, we know that the map y(-, t) : B — > B t is one-to-one, 
onto and its differential does not degenerate. That is, it is a diffeomorphism. 

Then we show that any two geodesies in the set {y(p, [0, T]) : p € B} are equal if they 
intersect. Suppose that there exists distinct points Pi, P2 G B and positive times ti, t 2 in 
(0, T) such that y(pi, ti) = y(p2, tz)- Then by the argument in the precedent paragraph 
we know ti ^ t2, say ti < t2. Then the piecewise smooth curve y(pi, [0, ti])Uy(p2, \Xz, T]) 
connecting B and B' has length ti + (T — tj) < T. Contradiction. 

The statements in the last two paragraphs shows that the map 

y(-, •) : [0, T] x B -> M, (p,t)^y(p,t) 

is a diffeomorphism. Moreover, since G leaves each hypersurface B t invariant and acts iso- 
metrically and effectively on it, B t is a sphere with constant sectional curvature. Therefore, 
the Riemannian metric gM of M can be written by gjvi = dt 2 + f 2 (t) g S n-i for some positive 
smooth function f(t) on [0, T]. □ 

Lemma 3.3 // 3M is connected, then 3M must be isometric to the unit sphere S n_1 . 

Proof. Suppose that 3M is isometric to the real projective space RP n_1 with the constant 
sectional curvature and n > 3. Then n should be even since RP even does not bound 
by the unoriented cobordism theory (see pp. 52-53 in Milnor-Stasheff |8J). Denote by G 
the identity component of the isometry group of M. Since dim G = n(n — l)/2, G is 
isomorphic to SO(n)/{±1} and its action on the boundary RP n_1 is induced by the linear 
action of SO(n) on S n_1 . Moreover, the isotropy subgroup H := G p at each point p on 
3M is isomorphic to SO(n — 1 ). Following the proof of Lemma 3.1, the G action on M has 
principal orbits of type G/H = RP n_1 . Denote by M(H) the union of principal orbits. Then 
the orbit space M(H)/G is diffeomorphic to the interval [0, 1). Since M(H)/G is dense in 
the total orbit space M/G, for the G action on M, there exists only one orbit G/J other 
than the principal ones. We call the orbit G/J the exceptional orbit. It is this orbit G/J 
that corresponds to the endpoint 1 of the orbit space M/G. Since H can be thought of as a 
proper subgroup of J in the sense of conjugacy, the Lie algebra Z of J contains a subalgebra 
isomorphic to so(n — J). Simple computation shows that Z is isomorphic to either so(n) or 
so(n — 1 ). 

Case 1 If Z is so(n), then J equals G. This means that topologically M is the cone of 
RP n_1 , which does not have the structure of a manifold. Actually, the cone of RP n_1 is 
homeomorphic to the orbifold D n /{±1} the boundary RP n_1 . Contradiction. 

Case 2 If Z is isomorphic to so(n— 1), then the exception orbit G/J is a submanifold of 
codimension 1 in M. So the G action on G/J is also effective. Since G attains the largest- 
possible dimension n(n — 1)/2, G/J is isometric to RP^ -1 by Fact 1.3, which implies that 



G/J is also a principal orbit. Contradiction. 



□ 



Lemma 3.4 If dM is isometric to the sphere S , then M is homeomorphic to either 
D n or RP n \U, where U is an n- dimensional open disk such that its closure U in RP n 
is homeomorphic to the closed unit ball B n c R n . 

Proof. We use the notation in the proof of Lemmas 3.1-3. Remember that G = SO(n) 
and H = SO(n — 1). By the similar argument in the proof of Lemma 3.3, there exists only- 
one orbit G/J (called the exceptional orbit) other than the principal orbits among all the 
orbits of the G action on M. Also by the argument in the proof of Lemma 3.3, we know 
that either J is SO(n) or J contains a subgroup of finite index and isomorphic to SO(n— 1). 

Case 1 If J is SO(n), then M is homeomorphic to the closed unit disk. 

Case 2 If SO(n — 1) is a subgroup of J of finite index, then the exceptional orbit G/J 
is a submanifold of codimension 1 in M. Hence G acts effectively on G/J. Since 3M is 
connected, G/J has to be isometric to RP n_1 by Fact 1.3. Therefore, M is homeomorphic 
to the mapping cone S n_1 x [0, 1]/ ~, where the equivalent relation ~ on S n_1 x [0, 1] is 
defined by (x, 1) ~ (— x, 1) (see p. 13 for the concept of mapping cone in p. 13 Hatcher [3]). 
Clearly M is also homeomorphic to the punctured real prjective space RP n \U. □ 

Lemma 3.5 // M is homeomorphic to the n- dimensional closed disk D n , then there 
exists a point O in the interior of M and a positive number R > such that the 
exponential map exp Q at O is a diffeomorphism of the closed ball centered at the 
origin G TqM and of radius R in TqM onto M. Moreover, the Riemannian metric 
9m of M is rotationally symmetric with respect to O so that it can be expressed by 

g M = dt 2 + cp 2 (t) g S n-i , 

where the function cp : (0, R] — > (0, oo) is smooth, cp(0) = 0, and 

<p(«»w0(0) =0, cp(0) = 1. 

Proof. We use the notation in the proof of Theorem 2.1 and Lemma 3.4. Denote by B 
the boundary 9M. We know that B is a sphere of constant sectional curvature. By the 
proof of Lemma 3.4, there exists a unique fixed point O G Int(M) of the G action on M. 
The G action on the tangent space TqM at O gives an isomorphism of G onto the special 
orthogonal transformation group SO (TqM) of the Euclidean space TqM. We denote by 
B(0, r) the set of tangent vectors of length < r at O, and by S(0, r) the set of tangent 
vectors of length r at O. We also denote by S(r) the set of points in M with distance r to 
O, by exp the exponential map at O. 

Choose a point p G B such that d(0, p) = d(0, B) =: R. Then there exists a unit 
tangent vector Vo at O such that p = exp (RVo), and the geodesic exp (tVo), t G [0, R], is 
perpendicular to B at p. Choose an arbitrary unit tangent vector V at O. We claim that 
the geodesic exp(tV), t > 0, meets the bounadry B perpendicularly at time R. Actually, 
choosing an isometry a G G such that the differential da at O maps V to V, we find that 
the geodesic exp Q (tV), t G [0, R], is the image of the one exp (tVo), t G [0, R], under the 



isometry a. Since G acts transitively on B, for each point p € B, there exists V G S(0, 1 ) such 
that exp (RV) = p. By the uniqueness of the geodesic perpendicular to B at a given point, 
we find that if exp (RV) = exp (RW) for any two vectors V, We S(0, 1 ), then V = W. Hence 
exp : S(0, R) — > S(R) is a smooth bijection and there is no cut point of O in the interior of 
M. So, exp gives a diffeomorphism of B(0, R) onto Int(M). To prove this diffeomorphism 
can extend to the boundary, by the Gauss lemma, we need only to show that the restriction 
of exp to S(0, R) is a diffeomorphism onto S(R) = B. 

Remember that G acts on both S(0, R) and S(R). Moreover, the exponential map exp : 
S(0, R) — > S(R) is an equivariant smooth bijection map with respect to the G actions. By 
the Sard theorem, there exists a regular value of exp \s[o, R]- O n the other hand, by the 
equivariant property, all points of S(R) are regular values. That is, the map exp : S(0, R) — > 
S(R) is a diffeomorphism. 

The similar statement holds for each (n — 1) dimensional sphere S(r), < r < R, in M. 
So the metric gM of M has rotational symmetry with respect to O. The expression of qm. 
follows from the argument in pp. 12-13 in Petersen [Qj. □ 

Remark 3.1 We can list closed geodesic balls with suitable radii in the three spaces R n , S n 
and H n as concrete examples of the manifold M in Lemma 3.5. Simultaneously, the geodesic 
annuli of these three spaces form examples of the manifold in Lemma 3.4. The manifold 
M in our consideration need not have constant sectional curvature, whose curvature can be 
computed explicitly in terms of the function f (see pp. 65-68 in Petersen [9]). Because of 
the large symmetry on them, this class of manifolds, including geodesic balls in R n , S n 
and H n , may be thought of as the simplest class of compact Riemannian manifolds with 
boundary. 

Lemma 3.6 We use the notation in the proof of Lemma 3.4- Suppose that M is home- 
omorphic to RP n \U. Then we can find a Riemannian manifold M/ = S n x [— T/2, T/2] 
endowed with the metric dt 2 + f (t) g$n-i, where f : [—T/2, T/2] — > (0, oo) is an even 
smooth function, and an involutive isometry (3 of M' defined by |3(x, t) = (— x, — t) 
such that M is the quotient space of M' by the group {1, (3}. Here —x means the an- 
tipodal point of x in S n ~ 1 . Of course, M is diffeomorphic to RP^XU. 

Proof. First of all, let us forget the Riemannian metric on M. Consider a topological 
model of M — the mapping cone S n ~ 1 x [0, 1]/ ~. Remember that the equivalent relation 
~ means (x, 1) ~ (— x, 1), where x i— > — x is the deck transformation of the 2-fold covering 
S 1 ^ 1 -> RP n ~ 1 . Then M is the quotient of M' := S 1 ^ 1 x [0, 2] by the group generated by 
the involution (3 of M given by 

(3(x, t) = (-x,2-t). 

Then we endow M' with the induced Riemannian metric from M. Since each isometry of 
M can be lifted to two isometries of M', M' also satisfies the condition of Theorem 1.3. By 
Lemma 3.2, there exists a positive number T and a smooth function f : [—T/2, T/2] — > (0, oo) 
such that M' is diffeomorphic to S n_1 x [—T/2, T/2] and the metric gM' is given by 

g M , = dt 2 + f 2 (t)g S n-i, 



where g$n-i is the standard metric on the unit sphere S n . On the other hand, since the 
deck transform (3 : M' — > M', (x, t) = (— x, — t), is an isometry of M', we can see that — x 
is actually the antipodal point of x e S n ^ and f (t) is an even function. □ 

We finally complete the proof of Theorem 1.1 by combining all the lemmas in this 
section. □ 



4 Proof of Theorem 1.2 



Let M be a Riemannian manifold satisfying the condition of Theorem 1.2 in this section. 
Let B be a compact component of 3M. By Proposition 12.21 and Fact 1.3, the isometry group 
1(B) of B has the largest-possible dimension u(n— 1)/2, so B is isometric to either S n_1 or 
RP n_1 with constant sectional curvature. Suppose the former case holds. Then G := I°(M) 
is isomorphic to SO(n) and the isotropy subgroup G x at each point x e B is isomorphic to 
H := SO(n — 1). By the same argument in the proof of Lemma 3.1, the G action on M 
has principal orbit type G/H and 3M has at most two components. We claim that 3M is 
connected, i.e. 3M = B. Actually, since the orbit space M(H)/G is a connected smooth 
1-manifold, whose boundary coincides with the orbit space 3M/G, M(H)/G is a compact 
interval if 3M have two components. Since the orbit space M(H)/G is dense in the total 
orbit space M/G, we have M(H) = M and M is diffeomorphic to S n_1 x [0, 1]. Contradict 
the noncompactness of M. By the same reason, we also have M(H) = M. We have the 
similar argument for the latter case. Summing up, we obtain 

Lemma 4.1 M is diffeomorphic to either S n ~ 1 x [0, 1) or RP n_1 x [0, 1). 

Lemma 4.2 

(1) Let M be complete. If M is diffeomorphic to S n_1 x [0, 1), then the metric qm of 
M can be expressed by 

g M = dt 2 + f 2 (t)g sn -,, 

where f : [0, oo) — > (0, oo) is a smooth function. The similar statement holds for M 
diffeomorphic to RP n_1 x [0, oo). 

(2) Let M be noncomplete. If M is diffeomorphic to S n_1 x [0, 1), then there exists a 
positive number T the metric qm of M can be expressed by 

g M =dt 2 + f 2 (t)g S u-i, 

where f : [0, T) — > (0, oo) is a smooth function. The similar statement holds for M 
diffeomorphic to RP n_1 x [0, 1). 

Proof. We only prove the case that M is diffeomorphic to S n_1 x [0, 1). We use the no- 
tation in the proof of Proposition 12.11 and Lemma 4.1. Choose an arbitrary point q in the 
interior of M such that the distance of q to the boundary B is D. Cutting M along the orbit 
through q of the G action, we obtain a compact part Mi diffeomorphic to S n_1 x [0, 1] and 
a noncompact part diffeomorphic to M, on both of which G acts isometrically. By the proof 
of Lemma 3.2, the map y(-, •) : B x [0, D] — > Mi is a diffeomorphism. Since q is arbitrary, 



there exists T e (0, oo] such that the map y(-, ■) : B x [0, T) -) M is a diffeomorphism. 
Moreover, T is oo if and only if M is complete. □ 



5 Proof of Theorem 1.3 

Since the compact transformation group theory cannot be applied directly to isometry 
groups of Riemannian manifolds with noncompact boundary, we need new ideas to clas- 
sify Riemannian manifolds with noncompact boundary whose isometry groups attain the 
maximal dimension. 

Denote by and Q^, the identity components of the isometry groups of R k and H k , re- 
spectively. Remember that Q\_ is the identity component of 0(1 , k) and semisimple for each 
k > 2. However, Gk is the semidirect product of SO(k) and R k , and it is not semisimple for 
each k > 1 (see p. 5 and p. 77 in Petersen [9]). Let M be a Riemannian manifold satisfying 
the assumption of Theorem 1.5 through this section. By Proposition 12.21 and Fact 1.2, every 
component of 3M with the induced Riemannian metric from M is isometric to either R n 1 
or the (n— 1 )-dimensional complete and simply connected Riemannian manifold H n_1 (c) of 
constant sectional curvature c < 0. Note that all H n_1 (c)'s, c < 0, have the same isometry 
group isomorphic to the semidirect product of <? n -i and Z2. Suppose that a component of 
3M is isometric to R n . Then I°(M) is isomorphic to G n _i, which acts effectively and 
isometrically on each component of 3M. Hence, we find that each component of 3M should 
be isometric to R n_1 . The similar argument goes through if 3M has a component isometric 
to H n_1 (c) for some c < 0. 

Lemma 5.1. Each component of 3M is isometric to either R n_1 and H n_1 (c) for 
some c < 0. Moreover, the components of 3M are mutually isometric up to a scaling 
of metric. 

Lemma 5.2. We use the notation of Proposition \2.1\ Let B be a component of the 
boundary 3M and p an arbitrary point o/B. Let I be the maximal existence interval 
of the geodesic y(p, t) = exp p (tn p ) perpendicular to B at the initial point p. Then the 
map 

y:BxI^M, (q,t)^y(q,t) 

is well defined and gives a diffeomorphism of B x I onto M. Consequently, if I is a 
compact interval, then M is diffeomorphic to B x [0, 1]; if I is an interval open at the 
right endpoint, then M is diffeomorphic to B x [0, 1). 

Proof. Denote by G the identity component of I(M). Then G = G n _i if B is isometric to 
R n_1 , G = 9n-\ if B is isometric to H n ~ 1 (c) for some c < 0. We only prove the case where 
B is isometic to R n 1 . The other case can be proved similarly. 

Since G acts transitively on B, we can see that, for every point q € B, the geodesic 
y(q, t) perpendicular to B at the initial point q also has the maximal existence interval 
I. We claim that for any two distinct points p, q G B, the two geodesies {y(p, t) : t e 1} 
and {y(q, t) : t £ 1} does not intersect at a point x such that x = y(p, s) = y(q, s) for 



some s G I and d(x, p) = d(x, q) = s. Otherwise, we take an element a G G mapping p 
to q and the sequence of points p n = a n (p) goes to the infinity of B as n — > oo, which 
implies that the subgroup V generated by a is not precompact in G. On the other hand, 
since y(a(p), s) = a(y(p, s)), x is the fixed point of a. So the isotropy subgroup G x of the 
G action at x contains a non-precompact subgroup P. Contradict the compactness of the 
isotropy group G x . 

We claim that the subset B t = {y(p, t) : p € B} is a Riemannian submanifold isometric 
to R n 1 for each tel. By the equality <xoy(-, t) = y(-, t) o a for every a G G, B t is exactly 
an orbit of the G action, so it is a submanifold of M. Remeber that the map y(-, t) : B — > B t 
is surjective and G-equivariant. By the claim in the preceding paragraph, this map is one-to- 
one. Hence, it gives a diffeomorphism of B onto B t . Since G acts effectively and isometrically 
on B t , the claim follows from Fact 1.3. 

The left part of the proof is similarly to that of Lemma 3.2. There also holds that that 
for each (p, t) e B x I 

d (B, B t ) = d (y(p, t), B) = d (y(p, t), p) = t. 
And the geodesic {y(p, t) : t e 1} is perpendicular to B t at point y(p, t). □ 

The proof of Theorem 1.3 follows from Lemma 5.2. □ 
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